import s3fs
import xarray as xr
import hvplot.xarray
import geoviews as gv
import datashader as dsh
from holoviews.operation.datashader import rasterize
gv.output(size=300)Downsample zarr
xarray and dask with data from SMAP
Run this notebook
You can launch this notebook in VEDA JupyterHub by clicking the link below.
Launch in VEDA JupyterHub (requires access)
Learn more
Inside the Hub
This notebook was written on the VEDA JupyterHub and as such is designed to be run on a jupyterhub which is associated with an AWS IAM role which has been granted permissions to the VEDA data store via its bucket policy. The instance used provided 16GB of RAM.
See (VEDA Analytics JupyterHub Access)[https://nasa-impact.github.io/veda-docs/veda-jh-access.html] for information about how to gain access.
Outside the Hub
The data is in a protected bucket. Please request access by emailng aimee@developmentseed.org or alexandra@developmentseed.org and providing your affiliation, interest in or expected use of the dataset and an AWS IAM role or user Amazon Resource Name (ARN). The team will help you configure the cognito client.
You should then run:
%run -i 'cognito_login.py'
Approach
This notebook demonstrates 2 strategies for to subselect data from a Zarr dataset in order to visualize using the memory of a notebook.
- Downsample the temporal resolution of the data using
xarray.DataArray.resample - Coarsening the spatial aspect of the data using
xarray.DataArray.coarsen
A strategy for visualizing any large amount of data is Datashader which bins data into a fixed 2-D array. The call to rasterize ensures the use of the datashader library to bin the data.
About the data
The SMAP mission is an orbiting observatory that measures the amount of water in the surface soil everywhere on Earth.
Load libraries
Optional: Create and Scale a Dask Cluster
We create a separate Dask cluster to speed up reprojecting the data (and other potential computations which could be required and are parallelizable).
Note if you skip this cell you will still be using Dask, you’ll just be using the machine where you are running this notebook.
from dask_gateway import GatewayCluster, Gateway
gateway = Gateway()
clusters = gateway.list_clusters()
# connect to an existing cluster - this is useful when the kernel shutdown in the middle of an interactive session
if clusters:
cluster = gateway.connect(clusters[0].name)
else:
cluster = GatewayCluster(shutdown_on_close=True)
cluster.scale(16)
client = cluster.get_client()
clientOpen the dataset from S3
s3 = s3fs.S3FileSystem()
root = "veda-data-store-staging/EIS/zarr/SPL3SMP.zarr"
store = s3fs.S3Map(root=root, s3=s3)
ds = xr.open_zarr(store=store)
ds<xarray.Dataset>
Dimensions: (northing_m: 406, easting_m: 964,
datetime: 1679)
Coordinates:
* datetime (datetime) datetime64[ns] 2018-01-01 ... 2...
* easting_m (easting_m) float64 -1.735e+07 ... 1.735e+07
* northing_m (northing_m) float64 7.297e+06 ... -7.297e+06
Data variables: (12/26)
albedo (northing_m, easting_m, datetime) float32 dask.array<chunksize=(100, 100, 100), meta=np.ndarray>
albedo_pm (northing_m, easting_m, datetime) float32 dask.array<chunksize=(100, 100, 100), meta=np.ndarray>
bulk_density (northing_m, easting_m, datetime) float32 dask.array<chunksize=(100, 100, 100), meta=np.ndarray>
bulk_density_pm (northing_m, easting_m, datetime) float32 dask.array<chunksize=(100, 100, 100), meta=np.ndarray>
clay_fraction (northing_m, easting_m, datetime) float32 dask.array<chunksize=(100, 100, 100), meta=np.ndarray>
clay_fraction_pm (northing_m, easting_m, datetime) float32 dask.array<chunksize=(100, 100, 100), meta=np.ndarray>
... ...
static_water_body_fraction (northing_m, easting_m, datetime) float32 dask.array<chunksize=(100, 100, 100), meta=np.ndarray>
static_water_body_fraction_pm (northing_m, easting_m, datetime) float32 dask.array<chunksize=(100, 100, 100), meta=np.ndarray>
surface_flag (northing_m, easting_m, datetime) float32 dask.array<chunksize=(100, 100, 100), meta=np.ndarray>
surface_flag_pm (northing_m, easting_m, datetime) float32 dask.array<chunksize=(100, 100, 100), meta=np.ndarray>
surface_temperature (northing_m, easting_m, datetime) float32 dask.array<chunksize=(100, 100, 100), meta=np.ndarray>
surface_temperature_pm (northing_m, easting_m, datetime) float32 dask.array<chunksize=(100, 100, 100), meta=np.ndarray>